Bagging Down-Weights Leverage Points

Bagging is a procedure averaging estimators trained on bootstrap samples. Numerous experiments have shown that bagged estimates often yield better results than the original predictor, and several explanations have been given to account for this gain. However, six years from its introduction, bagging is still not fully understood. Most explanations given until now are based on global properties of the estimates. Here, we focus on the local effects on leverage points, i.e. on observations whose fitted values are largely determined by the corresponding response values. These points are shown experimentally to be down-weighted by bagging. The performance of the bagged estimate depends on the goodness of these points for the original estimator. Illustrative examples findings are supported by the study of smoothing matrix, and their consequences are discussed.